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ABSTRACT 


iitewatrcitpeciatian Approach, a proposed methodology 
oeeoest nid” sand implementing Vecision Support Systems 


(DSS), attempts to integrate modular design and 


adaptive design. The approach is based on decomposing 
mie proposed system’s tasks nem stile ured and 
mews tructured modules, 6valucauwanme “thie difftficulty of 


implementing each module, and utilizing the estimated 
@itficulty and the priority of each module to determine 
whe best development sequence. The £easitbadlaty sor 


taicis teliable “and accurate predictions of implemen- 


mab lon difficulty, a key requisite, was previously not 
verified. Utiteumieskows Dresecues a Gilscussion of the 
Archipelagian Approach and aneeeecinott Toate «SLCUdY “OL 


ZMiisaGts boats potentially could be used to predict 
miplementation difficulty. Eme study concludes that 
MmiwemoOumr tc scl On, haetrOrs "considered exhibit sufficient 
Gewese ty eand Validity as predictors to confirm the 


Praoility of the approach. 
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A. GENERAL BACKGROUND 

The Archipelagian Approach, a proposed methodology 
Boece sioenineg and amplementing Decision Support Systems 
(DSS), attempts to integrate and capture the advantages 
free th modular design and adaptive design. A key step 
requires the DSS builder to accurately estimate module 
Seeonornonaotli ty, a predictiow ofr the difficulty of 
implementing each module in the planned project. i eS 
reliable, valid prediction measure is feasible, then 
the proposed methodology becomes merely an academic 
Dos Ccloscmm~mir noua? UNCELOnal application: Thistechies its 
mreesents a discussion of the Archipelagian Approach, 
mia ateempiricalvevaluation of potential factors that 


fae OC Utilized to Estimate module accomplishability. 


Be OBJECTIVE 

Mimo yCCULyVe Oot this thesis is to evaluate the 
mieabtilaity of the Archipelagian” Approach requirement to 
maecdi ct module accomplisnhability. The study identifies 
Mmessible factors or variables that could serve as 
Mmeedictors of accomplishability, and assesses’ the 
meliability and validity of the estimates made when 


these factors are utilized to evaluate sample modules. 
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C. RESEARCH QUESTIONS 

Four main questions “atew J4¢e1eoceur (1) Wha 
factors should be used to estimate accomp! !shad vie 
(2) What is the inter-rater reliability for eacem 
factor? (3) How valid are the implementation feasi- 
bility predictions made using each factor? (4) What 
conclusions can be drawn about the viabdiliity ere 


Archipelagian methodology? 


Dy. SCOPE AND LIMITATIONS 

In acd 1 tal On to the implémentation diiiTecuie 
prediction, or Accomplishability = Pactor CAF), “alae 
Archipelagian Approach utilizes an Imperative Face 
(IF) to express the priority associated with cac@ 
module , and a Development Priority Factor (DPF) to 
determine module development sequence. The IF andre 
are explained in the background discussion, DU Gea 
addressed in the empirical portion of the study. 

The Archipelagian Approach is intended for Use 
Decision Support System (DSS) builders. Responses from 
practitioners would have been preferred for G@valUatine 
the reliability and validity of potential accompa. 
abllity measures. However. to facilitate correcta. 
data, a survey of graduate students in the iit 
quarter of the Computer Systems Management (367) 


curriculum at the Naval Postgraduate School was uecae 
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BE. LITERATURE REVIEW AND METHODOLOGY 

This research is based on issues raised in a paper 
veer. xX. bail and [.. Kk. SivasanKaran of the Naval 
Beaeteoraduate School faculty titled "Integrating Modular 
Pesic@m@reenn Adaptive Desisn ine pss Prototyping: An 
Archipelagian Approach" Ligh e ieletalies| a) tmcemacCOnCept is 
proposed [Ref. 1]. A summary of the approach appears 
mec hapter [Two of this thesis. 

liccmmicthod@etwocy Sor conducting the study included 
mour Steps. (1) Review of literature to identify 
factors that DSS researchers postulate could affect 
pecomplishability. (2) eDesieon Sol aw questfonnaire 
Containing narrative des cigtoiskons Meotmmmodules for 
mespOndents to evaluate using the selected factors. 
Memnadministration of the questionnaire to a group of 
[i> Sraduate students. (4) Statistical analysis of the 
results using Minitab Release 5.1 running under VM/CMS 


em an IBM 3035 computer. 


F. SUMMARY OF FINDINGS 

Pecoiteoosstp he module  siimplementation difficulty 
mmeedictors or factors were selected for the study. The 
factors were Task Complexity, Task Programmability, 
Mask Structure, Module Size, Hoot AVailabi lity, “value 
mea cement , task “Analyzability, and, ~Completion lame. 


moe rirst five factors listed exhibited significantly 
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higher 1nter=-raverwee! ella Oommen a The validity 6 ia 
factors as@predierers of accomplishability™ Uusitiae 
estimates made by individual raters was disappointing. 
Ow june wever, the results using the group means were 
highly accurate, demonstrating that predictl1on == 
accomplishability is practical if aggregate judgements 
on the factors are used. The high correlation coef- 
ficients among the high-reliability factors imply ce 
they are 'flargely ne@gundanin:. The study did not underme 
take a factor analysis to determine which of the 


factors should be eliminated. 


G. ORGANIZATION OP=stuUDy 

Chapter Two presents the theoretical background for 
the study, including discussions of modular design, 
adaptive design, and the Archipelagian Approach 
Chapter Three describes the study methodology, ifecusim 
on the construction of the questionnaire, and eu 
marizes the collected data. Chapter Four contains tm. 
analysis results and possible interpretations. The 
closing chapter presents conclusions, recomnenada ts 


and questions for further researcn. 


1@ 


Il. BACKGROUND AND THEORETICAL FRAMEWORK 


A. MODULAR DESIGN 

Docubate Of se scLructured systems desisneis amwdisci-— 
plined methodology for computer system design that 
Mosc meahuatuemph “GLoOsavold the high cost andw#poor 
maintainability associated With earilicrap cOlegware 
foe LOopMenia, methods. Vitti mo Guia wea es 1 or; aie eae, 
Somplex systems are partitioned into simple, inde- 
Pid niLackioOxymemogubes organized into Hierarchies 
Peano) komnOr seonlpuiber implementation . Liewime DHhOGgolLesy 
ime Ludes~ ocraphic LO© | omnOnmes CAaSy. COMMUNI Cation ~or 
specifications and design results, a set of strategies 
Maiemocveloapunc sdesien solutions, and a set of criteria 
ine Voluabtane  —thessquality of the resulting design 
Solution. |Ref. 2] 

Scoot madvanbacTesSwecaneabe realized through the use 
meamecwlar bechniques. First,scomplex systems are more 
Basil Voungeratood when partitioned into simple modules. 
Beeond ,—development is more rapid because modules can 
Pec oroCmuatcd Lested in parallel and reused in other 
proj ects. Piet nc Araphileoe tools sof modular design 
mEevide  £oOOd system documentation. FPOuUrti,..mecdular 
systems are more reliable, easier to modify, and less 


expensive to maintain. [Ref. 3] 
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Be. ADAPTIVE DEStean 

Despite the advantages, modular design has gen- 
erally not been applied to the development 61 Dec. a. 
SUPDOrt ovotems - The poorly-structured nature ofa 
tasks, and the evolutionary nature of the DSS environ- 
ment, preclude the complete, one-Stép Specit teatro 
functional requirements for the system in advance [Ref. 
Lae Instead, researchers advocate the adaptive desia@e 
strategy. Adaptive design is an iterative technique: 
which the final system emerges through a seritesuen 
DroOovtotypes. The initial prototype, produced quires, 
and on a small scale, represents computer-based supperg 
of a limited subpr epilenr Through interaction Withee... 
initial sys ten: users develop new perceptions am 
insights which stimulate the need for new function. 
these new requirements are incorporated into tie meas 
generation prototype by the wader. This interactren 
between users and builders continués untila@ savici4e. 
tory final system is completed. [Ref. 5] 

Adaptive design provides the flexibility necess2g 
to approach the automation of poorly-structurcd eta 
However, the strategy treats the entire systema 
poorly-structured, leading to unnecessary and Cos ta 
interactions between the users and builders over (oae 
defined functions [Ref. 6]. In addition, prototype 


ignores the potential benefits of modular) decree 
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Hither method can lead the DSS development team to 
Wweste cime, effort, and resources on a project that is 


teclnmactely abandoned because some key feature cannot be 


implemented . 


C. THE ARCHIPELAGIAN APPROACH 

Licwmarchipelacian “Approach to Pease DroOvuoupypins 
PanocmptS to secure the advantages of the modular design 
mernod while maintaining the flexibility of the 
adaptive design strategy. ENG wapprOacn CONSISts Of 
four basic steps [Ref. 1]. 

1. Step One 

DEcemMpose bite proposed DSS into as many 

functional subsystems as possible, and decompose each 
epsystem into its component modules. This results in 
dividing a conten OOrlye-sturuce FbUred problem into 
"islands" of beth Strmuecuulmed suGuo. sil 1h ILS sig qibicre bngterel 
subproblems (and provides the inspiration for the name 
"archipelagian"). 

Zee) Iwo 

Compute an Accomplishability Factor (AF) for 

Seach module. ifeerhe initial paper, the Al is conceived 
meme function of Perceived Task Structure and Tool 
Pvaillability, and is expressed ona scale from zero 
(very low) to one (very high). Modules with a very 


Pen AB would be relatively easy to implement and are 


ee, 


Sultable yy for structured design and implemneniacae. 
techniques, while modules with a lower AF entail more 
risk and probably require implementation throushi ea 
adaptive ,desilen me phode or. 

5. step Three 


Specify an Imperative Factor (IF) for each 


module. This factor allows the incorporation of User 
priorities and implementation sequence cons trams. 
into the development. stravecuwe Modules representa 


functions that the users desire the mosty” or Utama 
be completed as a prerequisite to building somewerwn 
modules, will have an IF close to 4emeer Modules thaw 
can wait are assigned an IF closer to one. 

4. Die pase Our 

Compute a Development Priority Factor (DPF) 

for each module to aid the DSS Builder in determina 
module development sequence that will progressively 
reduce Ore eC tlcia. The DPF is the product of "thew as 
and the IF. Since the more risky modules will have a 
low DPF as a result of their low AF, @enplenmen ae 
modules in order from low to high DPF will mini 
wasted effort if the project is cancelled beCcanc~eme. 
inability to implement a risky module. 

The key to the Archipelagian Approach is theweceaw 
step, computing the Accomplishability™ Pac tery) tage 


ai el Dalge Be reliable method of predicting ™the likeli 
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of successfully implementing each module, the approach 
Saunoet» OC ~ ab iaed- thesswemainder of this study 


addresses the accomplishability prediction issue. 


i 


III. METHOD@ROGY Alper 


A. QUESTIONNAIRE CONSTRUCTION 

The survey questionnaire was developed throusiaae 
three-step process. (1) Potential variables or fac theme 
that could affect accomplishability were selected. 
study. (2) A precise definition and rating scale wae 
drafted for each selected) tac ter (3) =Narratagae 
descriptions of sample program modules were prepared 
for respondents to evaluate using the specified facuen 
definitions and rating scales. The complete quéeéstaiea. 
naire is included as the Appendi terra: tio 

Potential factors were evaluated against three Gra 
teria. First, there had to be at léast an i173 
sense that the factor was likely to affect the abxeee 
to implement pregramemedules - Second, the factor iad 
to have potential variability across different modules 
in the same project, not only across diiferent eo. ieee 
or development teams. Third, the factor had tome 
capable of expression in the form of a_escale or 
standards that individuals could use in makings ac. 
ments? “on: thes modules: The factors selected View 
inclusion in the questionnaire are listed in Tate 
below, along with the abbreviations that will bene 


in the data summary and analysis portions of the 


is 


ere (e @ne ae. Mm tOM™m~EO su neme Ccloht factors, Estimated 
Accomplishability was incorporated to re pile sei uae a 


summary judgement. 


POTENTIAL FACTORS 


TABLE 1 

Factor ppDrevlabtrTorm 
Task Complexity Gime bx 
Task Programmability jouer: 
Task Structure Struc 
Task Analyzability analy 
Vaaue Judgcemnent value 
VeGile AY a) baba la ty tool 
Module Size size 
Completaion Time 16 abionke, 
Estimated Accomplishability estwace 


The definition prepared for each factor emphasized 
msc applicability to the implementation of tasks at the 
meduile level. Withb ic eoxne cD bron  Ommeonpletion “lime , 
eaen factor was assigned arating scale between Zero 
and one, with five possible values for respondents to 


ZNVOOSEe . MEECecOr!DtLTon fer cach ~value provided a 
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standard that the sample modules could be compared 
against. Respondents were asked to estimate Completion 
Time for sample modules directly in man-hours. 
complete set of definitions and rating scale descragae 
tions is listed in the Appendix. 

The twelve sample modules for the questionneaww. 
were selected with the goal of covering a  Varictyeees 
Situations and implementation difficulties. To alte 
estimation of the actual accomplishability, module 
from existing DSS projects were utilized. Modules tem 
[Ref. 7], two through five [Ref. 8], and nine through 
twelve [Ref. 9] originated in previous Naval Post- 
graduate School thesis projects; modules sevén and 
eight were inspired by commercial products [Ref. 10]. 
The description for module six was deliberately drat 
to represent a task that is currently impossible wait 
COMpPUTEY \~presran. Tiare actual accomplishabi lia 
assigned to each module represents a judgement by this 
researcher based on the degree to which “Che nea 
MmEEets 25S Staced wurpose. and on the degree of imple- 
mentation difficulty reported by the module’ s™aut@a@ 
The values for actual accomplishability appear in the 
"actual" column of Table 2 in the Collected 


SCCLMonMOot this reper: 


i 


B. SAMPLE SIZE AND DEMOGRAPHICS 

The questionnaire was administered to 47 students 
Ade miiavcunrOoscberaduare=oecnOOl on December 10 and 11, 
| Sister? Mi Otte Mie liiticGdm | Ormsawere incomplete, 
leaving 45 usable responses. Of the 45 respondents, 39 
Veegemesbuagcnis ~i1n thewsetrifth —~quarter of the Computer 
Systems Management (367) curriculum, three were in the 
Porirdwmduarter Of the 46/7 curriculum, and three were 
from the Telecommunications System Management (620) 
Sortculum. Only GC€ight of the respondents reported any 
experience with computer software design or development 


outside of their course work. 


C. COLLECTED DATA 

Tine table on the following page lists the sample 
mean (m) and sample standard deviation (s) of the 
moiues selected by all respondents for each factor in 
rating each module. Completion Time is expressed in 
MmmGiheads Of —Man—-hours; all other factors are in terms 


of the zero to one scale. 


ie 


module 


ON anv FP WD - 


No Sr AD 


module 


OANIAU FWD 


ho = Sr 4 


cmp1x 
m s 
(O04 e.22 
Zee ee 
pea e | 
eee WO 
eel Peal 
be -ewew 
264.420 
S55 Seng 
<4 «20 
48 (see 
SO25 wee 
“967% 22 
tool 
m s 
eee ..135 
Pe a) mel 
eta "25 
«56. We24 
Oo} wee 4: 
15. ee. 
-60° 82718 
Sho- ae 
S00. pellet 
e000 oaee 
Owe eee 
94 .15 


COLLECTED DATA SUMMARY 


TABLE 2 
prog struc 
m 6s m os 
85 > se 219 seal 
709 220 (Ome 2 
269° to 10. alee 
25407 217 52 es 
ae > Da | 226-220 
oo ee 2 | 40> eree 
Pe Tee el) Cee) 
OC ae dS -64 .20 
-O2. Vee 4 Soe) Sle 
2007 "see So. ce 
ee ad 168 .20 
2960 12 2960 21:0 
Size time 
m s m s 
02am 2D le) 29 
PEO see Geib, V2 
waa eae, at AO 
-29 218 10.6 516-41 
eee he 85.4 2s 
-19° 216 76.@ Aile2 
D9 «15 teal ia ees, 
else. is lle Oe Fa oa 
258-2 iro Sz (ope 
41 oe \ 1,0 eee ol 
ne Oeines| 6.4 14.4 
.82 .20 0.8 ey 
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analy 
m Ss 
AT nese 
soo wee 
; oO leeee 
54 .24 
AO 5 tae 
PS js ela) 
OO mee 
<o eeee 
65) eee 
One .2 1 
53° See 
35 ee 
est.acc 
m Ss 
6S 2. 
OO a2 
-OG° sie 
Dees 
<) pae le 
oo Smee 
SIS ee ch, 
OT ee ie 
-79 «23.19 
58 .24 
-64 .24 
94 .12 


value 


73 
7 to 
73 
. 70 
49 
53 
etl 
66 
a 
59 
53 
. 70 


|” 


28 
ones 
29 
<ee 
2a) 
rh 
eal 
29 
~24 
ea 
i26 
~2o 


actual 


—-aqoaagageaeeedeadcr®?) & 


oe 
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oe 
50 
soe 
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50 
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ot 
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IV. DATA ANALYSIS AND INTERPRETATION 


Penematravino The pmaaeumecality of predicting module 
accomplishability requires finding a measure that is 
bewa reliable “and valid. Vemavaqid redundancy, the 
factors comprising the measure should be as independent 
as possible. This analysis addresses these issues in 


sequence. 


A. INTER-RATER RELIABILITY 

The reliability of a measure refers to the degree 
MomwiltChimmunmesresults of measurement are free of error. 
Miecbhis Study, the inter-rater reliability of a given 
factor represents the degree to which the different 
questionnaire respondents selected the same value for 
the factor when rating the same module. Pape |) pe low 
lists two indicators of relative inter-rater agreement 
mor @C€ach factor. The pooled standard deviation is the 
square root of the mean squared error for all respond- 
aunoeetiade all modules; it is expressed in the same units 
as the scale for each factor. Eta’ is equal to one 
minus the relative error, where relative error is the 
error variance divided by the total variance [Ref. 11]. 
This statistic would equal one if all raters were in 


complete agreement on every module, and would equal 


Z| 


zero if all variance between the ratings for ait terems, 


modules was due solely to differences between raters. 


INTER-RATER REDIABILC TS 


TABLE 3 
LAG Or Dee lems eta 
prog Osis @.47 
est.acc @.186 @.44 
St rue 0.195 Q@.41 
Se 0.197 @.44 
emp i @.204 @.46 
Teo. 0 22a DoS 
value @.243 Fy lla 
analy 0.265 @.@04 
elie Se7Ge @.@9 


Interpreting these results, the reliability for the 
last three factors (Value Judgement, Task Analyz- 
ability, and Completion Time) is clearly much lower 
than for the others; these three are consequently much 
l6ss usciu Lease predke tomer Thee ecmiain aa issuc ae 
whether the reliability of the other factors is) fag@ 


enough for them to be utilized ina practical Meas 


Fis fee 


Bea econ isha. Since whe Indicated réliabilities 
of 358% to 47% seem low. Low inter-rater reliability 
Gan bem wobteriputed to eCGeither low variability of tasks, 
Semin eh vearliablilityeotf raters. Reviewing Table 2, low 
Pasarela DL tater Se DOLeCntbtally a ®problem only for Task 
AMealyzability and Value Judgement. Ome e ic wIceilia ris ma 
MmICLOrs, Tfrater variability must account for the low 
meklabilities. 

IwWOmMDOltime reduce Vrmeme Significance of the low 
Para yenuUnoCcis. First, some varlatlion in factor 
hopmaes 1S expected because the factors in the ques- 
iPronmaire represent subjective judgments that are 
highly dependent on the individual rater’s knowledge, 
experience, and perceptions. For example, the variety 
fee DFOLsxTanminzg tools with which an individual is 
familiar can greatly influence the value chosen for 
heats Availability tor a given module, regardless of the 
tools that actually may exist. As another example, if 
Moeelnadivraual = Dercelves that a task 1s poorly struc- 
mmeea, then for that individual, Was Aue ake everoua Minis 
forucbured, cvem if some other rater may see a well- 
defined structure in the task. Piece lso: Likely that 
moc practical Imo Led ce evar led more between tne 
watecwmoomecomolering the questionnaire than it would 
meuween a “2roup of actual practitioners. The second 


point is that while the initial Archipelagian Approach 


a 


paper specified an interval scale f0r factors 
facilitate computation of the Accomplishability sa20 7. 
the important issue is the relative ranking Of jae 
modules. It is possible for raters to dittGreiengeee 
exact value assigned to each module, yet maintain the 
same relative ranking. As an illustration, modulewiaw. 
in this study clearly has a less structured (tase 
perform than module nine. The Task Structure raga 
assigned by the 45 questionnaire respondents varied 
from 9.90 to @.75 for module five, and from eoe23a 
1.99 for module nine, which results in relatively aaa 
inter-rater reliability. However, 88% of the raters 
marked module nine as more structured than they marked 
module five; 9% (four raters) had them even, and only 
2% (one person) thought module five was more struc- 
tured. Considering these Ypeimntee this researcher 
believes that the inter-rater reliabilities of 9a 
Programmability, Task Structure, Module Size, Taskeeanue 


plexity, and Tool Availability are not unusually low: 


Be. (VALID E LY 

The validity of a measure represents the degree to 
which it actually measures what it purports to Meacuiaee 
For this study, the coefficient of determination (r@) 
of the linear regression of the actual acconpiaa 


ability values on the evaluation factors provideerean 
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Siem oteciwmol. the Validity of the predictions. 
Table 4 below lists these coefficients for regressions 
Gt Doth the individual raters’ values and the aggregate 
emean) values. The factors marked with an asterisk are 
the ones determined in the previous section to have 


ZT whoveasihver—-racer reliability . 


REGRESSION COEFFICIENTS 


TABLE 4 
haciw Or, indiv. ré mean r¢< 
est.acc 55a 80.3% 
jee (yea 37.@ 79.6 
Sturue S225 ie 
Zool Bore 74.3 
Size ee | ecw 
Cmp ox au 6922 
*time 6.6 Hal 3 
*value aa oo. 7 
*analy a oral 


the regression results in Table 4 indicate that 
Saerew tie Validity of individual raters’ predictions 


was fairly low, the aggregate results were quite 


eS 


accurate, especially for the overall Estimaved feee7— 
plishabidlity ~Judeemeniu Only for module Six yea. 
intentionally "impossible" module, was the aggregate 
estimated accomplishability greatly different fren 
actual accomplishability “€see 2b ae-oe This was 
possibly due to a general reluctance for raters vee 
the low endpoint of the rating scale. Of Course je 
"actual" values, while based on more information @eme:® 
was available to the questionnaire respondents, still 
represent a judgement by the researcher and 9nie@eu. 
absolute standard. 

Further examination of Table 4 shows that no singde 
factor has an aggregate validity higher than ere 
summary Estimated Accomplishability. In addition. 
the simple regressions shown, multiple regressions were 
performed using combinations of two, three, and wigs 
factors. No combination had a coefficient of deteme 
mination significantly greater than the 8@.3% obtained 
using the raters’ Estimated Accomplishability. Si nee 
Every —Fespeonden. evaluated all factors for 6Veum, 
module, this study cannot determine whether equate 
accurate results could be obtained by simply asking 
raters to directly estimate accomplishability wivneue 
considering other factors, or if cCOnSsSiderationee ee 
separate factors contributes to the validity of the 


accomplishabDiamaty ratdiwes 


AS 


Mico l coc toOnmeieoll tT oma@isaplay only minor differen- 
tiation between the validity of Task Programmability, 
task obtructure, “ool Avatwabillity , Module Size, and 
iliac eo tenianya son Lhe Tresults do not provide a basis 
more selecting which factors “to include in the final 


Z~comperonabilitpy prediction measure. 


C. CORRELATION 

The correlation between two variables is a measure 
eee tne association between them. ner thas Study, 
merreclapion represents the degree to which two factors 
are measuring the same underlying variable that affects 
aeaeomplisnabilityes Table 5 on the following page lists 
ico samasem Dr Ooguct mMmonentweerrelation=cocitfificients for 
fies tactors correlated across both raters and tasks. 

Inspection of the correlation coefficients indi- 
cates that all five factors previously identified as 
having acceptable inter-rater reliabilities are, in 
general, highly correlated. Titi emediGica bes minat Ci Lier 
Mime lewels sarc in stact. interrelated, or that the 
Serrelation is a coincidence caused by chance corre- 
ifol@Gmset the factors in the sample modules in the 
questionnaire. Intuitively, this researcher feels that 
iiee oObructure, Task Programmability, and Task Com- 
Peeexity probably do overlap, but that Tool Availability 


should be independent. Module Size hee prebab 1 y 


Zt 


determined by the accomplishability, not vice-Vereas 
In any event, the correlation coefficients "dome 
provide a basis for selecting which factors to ince 
in the final accomplishability prediction measure mam, 


more than did the validity restive 


CORRELATION COEFFICIENTS 


Ji eile 5) 
est.acc Struc (pages ccnp ieee size value 


Struc @.766 

jOnek oye. Ot De oe Oe 

cmp1x 0.729 O71 oS eee 

tee | O.182 OR 75250 10s 655 

size 0.701 0.619 ©.626 50.658" 0-5 

value 0.3500 0.316 0.292 (022353 > Oiei0meee 199 


analy 2.105 @.@029 @.049 @.@066 @.046 @.032 0.099 
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V. CONCLUSIONS AND RECOMMENDATIONS 


A. CONCLUSIONS 


The analysis Of ete meGesUlGsr = rt om tlis Study 
bp Or ts Our mMalnmeconclusions . (ie) eeihe = dntéer=rater 
reliabilities Oe the summary judgement Estimated 


Accomplishability and the factors Task Programmability, 
Task Structure, Module Size, Task Complexity, and Tool 
Availability are high enough to allow them to be 
considered for use as predictors of module accomplish- 
aoali ty. HMitmveeiaiawer = -reliaerlities of Completion 
Time, Value Judgement, and Task Analyzability are not 
eee noOoush me tore, Luem Lo be considered for “use as 
predictors. (Zaawhisthke  thesvyalidity “of an individual’s 
Zeon lishability prediction is not likely to be high, 
the validity of predictions made using the aggregate 
results from a group of raters employing the high- 
Zermaorlity factors listed in Conclusion One should be 
eecellent. (3) In light of Conclusions One and Two, 
this study demonstrates that the basic Archipelagian 
Approach technique of predicting module implementation 
measibility is practical. A elicits correLavions 
among the high-reliability factors implies that they 
are interrelated, Somme should De possible Geryutrrize 


fewer than five factors to estimate accomplishability 
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without sacrificing predictive validity. Additiomew 
study is required to indicate which factors Canwee 


eliminated. 


B. RECOMMENDATIONS 

This researcher has three recommendations tomes 
as a mesultoeot petites Pcie (1) Practitioners wae 
decide to utilize the Archipelagian Approach semen 
employ the aggregate judgement of a group of Tratereuiae 
estimate module accomplishability instead of relying 
individual <resultse (2) The Archipelagian Approach 
authors should consider revising the AFP computapwee 
technique to utilize an ordinal scale instead "oie 
inter y al sealer. since relative differences between 
modules seem to be more important and can probabd ya 
estimated more reliably than ratings on a fixed scale. 
(3) Further research should be conducted on the 


approach. 


C. AREAS FOR FURTHER RESEARCH 

Inconclusively answered questions from thisws ear, 
provide topics for additionally ecea ec. (1) Can 
accomplishability be estimated in one Step; crea. 
multiple evaluation factors as used in this stu 
necessary? (2) Is the correlation observed between the 


high-reliability sfactors coincidenta® Or are "tiie 
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baetors LTeally related? An associated question is to 


Trem ne awed son tilew= aa on—-reliability factors can 


safely ljne eliminated Met waachwe. accomplishability 
Prediction measure Wie hnout Soact@iieineg “predictive 
walidity . (3) Would a similar questionnaire completed 


by a group of Decision Support System development 
Practitioners instead of by Computer Systems Management 
students result in higher inter-rater reliabilities? 
Vitekhe woe rea deen ot additonal research remains 
before the Archipelagian Approach will be completely 
validated, this initial study provides some evidence 
Miateetie COnCcCepte Of predicting the difficulty of 


Higeaemen tains proposed modules is practical. 


pil 
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APPENDIX 


DSS DEVELOPMENT QUESTIONNAIRE 


A. Ibu ever oyehb ve; 1c, 1efepal 

ites questionnaire is pamt of a research effort to 
evaluate a new proposal for designing and implementing 
Decision Support Systems (DSS). The proposal is based 
on the builder’s ability to decompose the proposed 
system into modules, and estimate in advance how easy 
Zomemodibes will be to construct. 

The following pages contain narrative descriptions 
of modules included in proposed DSS, and definitions 
ena rating scales for a set of factors for evaluating 
them. You will be assigning a score on each factor for 
woe mOouule. ASSUMe. that. an experienced programming 


team of average ability will be building the system. 


oe 


Be Factor Descraperems 


ilk flask Complesagmn 


a. — Det inten: The degree to which a items: 
involves a large number of variables, and the intrig@eae, 
of the interrelationships between variables. 

Db. Katine seale? 


1 


Q. 


Q. 


- 00 


LD 


5@ 


Wee, 


. OO 


ROUtInNe or utility task involyai 
essentially no variables. 

Simple task involving a few variables 
and uncomplicated interreVations iis 
Average task involving a few variables 
that may have involved or intricate 
interrelationshijps. or many variables 
but simple interdependenie rac 

Complex task involving many variables 
with intricate interdependencies, some 
of which are unknown. 

Virtually insurmountable task inva. 
ing a very large or infinite number ge. 
variables that are elaborately inva 
related, with many unknowns. 


eae Task Programmabi lity 


a.  -Deftini trons The degree to which a task can be 
or reduced to a step-by-step algorithm. 
D.—] Rating Seale: 


modeled, 


1 


Q 


Q. 


. OB 


LS. 


5@ 


Trivial task that can easily be perme 
formed with a few well-defined, simple 
Steps. 

ROoUrine task; the problem-solvane 
process may be lengthy or involved 
but an algorithm can bevdeVelapeds 


Part dads, non=procedural task “fier 
which an algorithm is probably aie 
possible, but which can be modems 


essentially completely. 

Non-procedural task that cannot Ge 
completely modeled, but which has some 
limited Saspeere that a model ean 
describe. 

Totally unprogrammable; every aspeer 
of the decision process involved is 
virtually impossible tosmode = 
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o.. MAS eorrue lure 


as 


Peta mer lOrik. The degree to which the variables 


involved in 


lo 


1.0@ 


One i> 


OO 


a task and the interrelationships between 
variables can be identified and precisely defined. 
Rating Scale: 


Ilak fealal L sve Semne CUCceC variables and 
Peotoaronshigs are ObVLOUS. 
Ser Ve wmuirc d : all variables and rela- 


tLonvcmses can be readily defined with 
limited effort. 

Partially unstructured; some variables 
that affect the task are hard to 


identify, or some inter-relationships 
are unclear. 

Mest a yainISs hie tur eC. 3 variables are 
hard to identify, and interrelation- 
ships between variables cannot be 
precisely defined. 

ovawky ne Unstructured; tie Variables 


Needed tewsolve the problem cannot be 
identified. 


aoe Task Analyzability 


a. 


lee 


Definition The degree to which analysis of 
the problem has the potential to identify alternative 
eyo Of Linding a solution. 

Rating Scale: 


1.@@ 


DEES 


0.50 


Dre > 


Unlimited analysis pete Eial; a 
correct solution can be reached ina 
ViilawvalL ly eintitt he Number ef ways. 
High analysis potential; a correct 
solution could be reached through any 
of a large number of approaches. 
Average analysis potential; any of a 
small number of approaches could lead 
Ome ecOmmeer sO LUG 1On . 

Limited analysis potential; only a 
very limited number of approaches are 
appropriate for the task. 

No» analysis potential; there is only 
one correct way to solve the task. 


bie. 


Bie Value suid e mens 


a. Definition: The worth or value of “havineue 


module included 


DOlmMt Of “Vtowe 


in the planned system from the wueemuae 


bo Katanesscacuse: 


lee) 2 


Sa. 


0.50 


Essential module; the system would be 
useless if the module was removed. 
Very desirable module; the usefulness 
of the system would be severely 
degraded without the module. 

Desirable module; the usefulness of 
the system would be somewhat degraded 
if the module were deleted. 

Optional module; the function prov ieee 
is nice to have, but not necessary for 
the system tG@ @emviser we 

Worthless “media the user would not 
even Notice Fae & Mies module were 
removed from the plans for the system. 


Oe Tool Availability 


a. DefLinivion- The degree to which appropriate 
hardware, programming languages, models, librawey 
software, CLC. exist for implementing the proposed 


module on a COmpUber system. 
Da  Ratgine “Scale: 


1.00 


The module can easily be implemented 
using any of a variety of available 
toOors. 


Tools are Known 0 6X1St; limited 
research would be necessary to select 
appropriate ones for the projceu. 
Tools probably exist; reséarch wile. 
necessary to identify them. some 
known tools would need minor modifi- 
cations in order to be used for eae 
Dr Oveci. 

some tools may exist; however, it is 
likely that some needed tools willl 
require major modifications or Necdiaas 
be developed from scratch. 

No tools exist and it is unlikely tmag 
any could be developed; the proj, cect 
not suitable for computer inp len ae 
tation. 
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oa Module Size 


a. IDYen@atietabimalenal iiemeseoebead lenath of tne module 
Mie iio ss orecodeo Usine a typical high-level programming 
faneuase (BASIC, PASCAL, FORTRAN, etc). 

loer ice t Ons ea Len: 


1.00 - Very small; less than 5@ lines of 
Cees = 

De FOS Sin ails: POO meleishioseOl Code. 

02.50 - Medium; 100-1000 lines of code. 

eas Large; 1@@@-5@00 lines of code. 

02.00 - Very large; more than S000 lines of 
coOcen 


a8 Complewmon Tame 


a. Detama tion : The estimated time that an 
average programming team would need to complete the 
detailed design and coding of a module. 

De. Rating Scale: Estimated time in man-hours. 


9. Estimated Accomplishability 


a Werinition: ite deeweoe = or cont @ence that a2 
focwlecw can be weimplemented .as part of .a computer 
program. 

Ratan socale: 


1.00 - Vey secasy ; Pe woOUlG Mer = or brivyral 
moagule that will require little 
work. 

OED an fa Rasy; an average programming effort 
Will be required, but few problems 
are anticipated. 

9.50. = Difficult; probably can be imple- 


HeMtGecoa,. bute soMmee= preliminary work is 
necessary to identify tools or work 
Oita mennod of attacking the 


problem. 

@.25 - Ve veer) 1 tetelgoladbt the module may not 
be possible to implement, and a 
major research effort will be 


necessary to develop tools before 

ViOmeewsen wnesmodgule can even start. 
0.00 - Impossible; jelGue adel bias erect iglenemlers 

implemented as a computer program. 


a 


C. Module" Deseret itons 


Circle the rating for each factor that, dies 


judgement, 1S Most appropriate. 


ils In a proposed system to assist with making assign- 
ments for military officers, this module will acca 
military ID number as an input, search a data Storeuien 
selected qualifying information about the officer "ma 
grade, specialty, etc), then search another data store 
to compile alist of all upcoming billet assicnien 


that the officer is qualitivea’ tomar 


a. Task Complexity 1.0 - .135 -— .50 - . 250 

b. Task Programmability 1.0 - .75 - .50 -— .25 =300 

Ge. Task" Strucuure 1.0 = .75 = S50 — 2255 

d. Task Analyzability 1.9 = ./15 — .50 — (253 

e. Value Judgement 1.0 - .75 - .50@ - .25 =e 

ff. Tool Availabe 1.90 = .75 = 1.50 — (2532 

Module size 1.0 - .75 = .530 - (253 
Completion Time (Estimated man-hours) 

i. Accomp lishability 1.9 - .15 - .50 —- (2530 

as In a proposed system to assist the Tactical Action 


OfLicer (TAO) on a surface ship in making tacuaeam 


decisions, this module will contain routines to allow 
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the user to input contact report data as it is received 


irom the ship’s sensors. 


(NOTE: In the questionnaires used for the survey, a 
Pmatinsescale. take the one on the previous page was 
PievTdCimUOLmecadel mouuke Go vallow responses to be 
recorded. in the interest of brevity, the rest of 


the scales are not reproduced in this Appendix. ) 


oe. In the same TAO system, this module will take 
Somtact report data trom the input module and maintain 
a contact database, dead-reckoning as necessary to 
Maintain updated positions on all contacts between 


meports . 


4. In the TAO system, this module will correlate 
information from the different sensors, and attempt to 
classify and identify contacts based on reported 
characteristics. It will also provide an ad-hoc query 


Capability into the contact database. 


2° 


ee In the TAO system, this module will searenetaee as 
a knowledge base of tactical directives and policies 
for required actions in the current tacticalee i Gaeae 
and search through a Knowledge base of stored his= 
torical conflicts for simi lar see 0battoen-- If a mage 
is found, the successful tactics used in the histor 
situation will be modified as appropriate t0 fp 


current situation and presented Gov the Goer ee ms 


Ce In the TAO system, this module will analyze the 
tactical situation independently of the historical 
knowledge-based module and work out the optimum tactics 


for the ship to folmoenw. 


ie In a system intended to assist managers with 
finance tak | odlanniin.. this module will use a flexible, 
English-like dialog to allow users to specify the 
Crateria Wor evaluating alternatives (net present 
value, return on sales, profit margin, payback period, 
etc) and select the type of financial problem to solve 
(merger/acquisition, project analysis, forecastime een 


cash-flow analysis). 


4Q 


Be Mimo hate ral planning system, this module will 


use computational routines such as goal programming, 


exponential smoothing, Ome mime ar regression to solve 
Mminancial problems. iicmmea Dp! Opriate technique 1s 
selected automatically, depending on the decision 


@riteria and type of problem previously selected by the 


Misc. 


9. In a proposed system designed to optimize the 
assignment of personnel to remote-site work teams for 
temporary projects, this module contains routines to 
Mpuis Update, and @€dit data about the available 


PaaoOlncolmeandmuac yoo positions to be filled. 


een tne personnel assignment system, this module 
will determine the "payoff" or value of assigning each 
momier tO Cach possible position. Since selection for 
a work team means family separation and difficult 
Beno Gondititons for the duration of the project, the 
goal is to spread the assignments as evenly as possible 


oer all available qualified personnel. 
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11. In the personnel assignment SYyYStem, this Moc 
will compute the optimum solution for the payorit Maui. 


utilizing the assignment method of linear programming 


12. In the personnel assignment system, this module 


will print a report listing the optimum assicune.. . 
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I¥V. Computer Background Information 

Since estimating how eet Palace lyk te a programming 
Peeyecer Will) See Ws a Subjective activity that varies 
Mien the Dackeroundeand experience of the person making 
the estimation, the following information is requested 
foal bow your Yrespemses tO De grouped with others that 
have a Similar background. Piliease  seircle or fill in 


ine appropriate choices. 


mee Curriculum: BOT Other (specify) 

wee Quarter: 5th Other (specify) 

3. Have you taken the following courses at NPS? 
a. Seruectured: Progcramning in Pascal YES NO 
eo sSOlLuware Economics YES NO 
lowe Software Design YES NO 
de. DeGaistvon Support SyYSTtenS YES NO 
e. Artificial Intelligence YES NO 

4. Do you have any experience in software design or 


development other than as a result of NPS classes? 
YES NO 
hag” SOdS Stabtewiew Many months, and briefly explain 


the nature of the work: 
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INITIAL DISTRIBUTIONS ae 


No. Copies 
Defense Technical Information Center 2 
Cameron Stav von 


Alexandria, Virginia 22504-6125 


Library, Code @142 2 
Naval Postgraduate School 
Monterey, California 93943-5002 


T.R. Sivasankaran, Codem@sasd 5, 
Dept. of Administrative sScmentce 

Naval Postgraduate School 

Monterey, California 93943-5000 


bh stephen G. Albers 2 


1251 Fuhrman Read 
Cincinnati, OCnio waa2i> 
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